.. _nova_orient: Nova.astrometry.net orientation =============================== Getting started: * you'll need an account on ``astrometry.net`` (for svn), and to have your ssh public key added to the authorized_keys for ``nova@oven.cosmo.fas.nyu.edu`` * can you ssh in as ``nova@oven.cosmo.fas.nyu.edu``? * can you ``svn checkout`` the Astrometry.net code? (see :ref:`svn_checkout`) Nova ---- As for the nova website code: if you svn checked out the Astrometry.net code, you should have a ``net`` directory -- that's where the server-side code that runs the nova.astrometry.net site lives. It's written in python using the Django framework, which provides nice web and database routines. We actually run three copies of the nova website: * supernova.astrometry.net -- "trunk", for testing and development * staging.astrometry.net -- test-flight area when we think we have a releaseable version. * nova.astrometry.net -- the production site. This runs off an "svn tagged" version -- not trunk The web sites are run via the apache web server. You probably won't have to deal with that side at all. Each one runs out of a directory within ``nova@oven``'s home directory. They're called (this may shock you), "supernova", "staging", and "nova". There is a server-side process that has to be running in order for the web site to do anything with images that people submit. That's called ``process_submissions.py`` (you can see the code in ``net/process_submissions.py``), and for each site we run it in a ``screen``. ``screen`` is an old-school utility that lets you create a "virtual terminal" that you can "attach" to and "detach" from. It's a handy way of keeping a program running on a server when you're not logged in. Anyway, if you're logged in as ``nova@oven`` you can do:: screen -ls to see the currently-running screens; :: nova@oven:~$ screen -ls There are screens on: 20579.supernova (03/18/2012 08:15:43 PM) (Detached) 27698.nova (03/14/2012 06:11:25 PM) (Detached) 2 Sockets in /var/run/screen/S-nova. which says there are two screens, ``supernova`` and ``nova``. You can "attach" or "rejoin" with:: screen -R supernova and you should see "process_submissions.py" doing its thing:: Checking for new Submissions Found 0 unstarted submissions Checking for UserImages without Jobs Found 0 userimages without Jobs Submissions running: 0 Jobs running: 0 anyway, to "detach" from that screen, do:: ctrl-a d Ok, so coming back to the code, let's find out where the data live. Each site has a ``settings`` file. So, as ``nova@oven``:: nova@oven:~$ cd supernova/net nova@oven:~/supernova/net$ ls -l settings.py lrwxrwxrwx 1 nova nova 21 2011-06-09 19:16 settings.py -> settings_supernova.py So in the ``supernova`` directory, ``settings.py`` is a symlink to ``settings_supernova.py`` . Let's look in there:: # settings_supernova.py from settings_common import * DATABASES['default']['NAME'] = 'an-supernova' LOGGING['loggers']['django.request']['level'] = 'WARN' SESSION_COOKIE_NAME = 'SupernovaAstrometrySession' # Used in process_submissions ssh_solver_config = 'an-supernova' sitename = 'supernova' oh, it hardly has anything. But at the top it imports everything from settigs_common.py. Looking there, we see:: ... WEB_DIR = os.path.dirname(astrometry.net.__file__) + '/' DATADIR = os.path.join(WEB_DIR, 'data') ... so it figures out the directory it is currently running in based on the location of the ``astrometry.net`` python package (``WEB_DIR``), which for supernova will be:: /home/nova/supernova/net and the ``DATADIR`` is that + ``data``. If you look in that ``data`` directory, you'll see:: nova@oven:~/supernova/net$ ls data 00 08 12 19 25 2c 34 3a 41 49 50 57 60 67 6f 74 7e 8a 91 9a a3 ad b4 bf c6 d4 da e0 eb f5 03 0b 14 1a 27 2e 36 3b 43 4a 52 59 62 6a 70 75 85 8b 92 9d a6 ae b5 c0 c7 d5 db e2 ef fa 04 0c 15 20 29 2f 37 3d 44 4b 53 5a 63 6b 71 76 86 8c 94 9e aa b0 b7 c3 ca d6 dc e4 f0 fb 06 0d 16 23 2a 32 38 3e 45 4c 55 5b 65 6c 72 79 87 8d 98 9f ab b1 b9 c4 cb d7 dd e6 f2 fd 07 11 18 24 2b 33 39 3f 46 4e 56 5e 66 6d 73 7a 89 8f 99 a2 ac b3 ba c5 d1 d8 de e9 f3 ff ok, a bunch of directories. What's in them? :: nova@oven:~/supernova/net$ find data/00 data/00 data/00/32 data/00/32/84 data/00/32/84/00328489cbdfce1a99ebbf1078c95669e39fa8a7 :: nova@oven:~/supernova/net$ file data/00/32/84/00328489cbdfce1a99ebbf1078c95669e39fa8a7 data/00/32/84/00328489cbdfce1a99ebbf1078c95669e39fa8a7: JPEG image data, JFIF standard 1.01 And check this out:: nova@oven:~/supernova/net$ sha1sum data/00/32/84/00328489cbdfce1a99ebbf1078c95669e39fa8a7 00328489cbdfce1a99ebbf1078c95669e39fa8a7 data/00/32/84/00328489cbdfce1a99ebbf1078c95669e39fa8a7 So the files are named according to a cryptographic hash of their contents (SHA-1), and sorted into subdirectories according to the first three pairs of hexadecimal digits:: AA/BB/CC/AABBCC.... (We sort them into subdirectories like that to avoid having a huge number of files in a single directory.) .. _web_local: Running on a local machine (your laptop, say) --------------------------------------------- Prerequisites: * `django `_ * `python-openid `_ * `django-openid-auth `_ * `South `_ * `PIL `_ * `sqlite3` * `simplejson `_ Once you've got the code checked out, you need to symlink one of the ``settings_*.py`` files to ``settings.py``:: ln -s settings_test.py settings.py for local testing, ``settings_test.py`` is the best bet, since it uses a local sqlite3 database rather than a real external database. Initialize the database:: python manage.py syncdb python manage.py migrate Then you can run the Django web server by running:: python manage.py runserver And then go to: http://localhost:8000