Sourcefinder is a BOINC based distributed computing project that aims to provide a means to test the effectiveness of various astronomical sourcefinding applications.
Sourcefinding applications are designed to process ‘cubes’ of data that contain radio sources and identify the location of the sources within the cube.
We’re currently testing the effectiveness of two different sourcefinding applications: Duchamp and SoFiA.
The effectiveness of a sourcefinding application can be determined by comparing the number of correct sources found within a cube to the number of incorrect sources found.
In order to provide a fair comparison between sourcefinding applications, we’re currently getting each application to process the same sections of a large cube known as the DINGO cube. DINGO is a simualated cube of data that contains a known catalogue of valid sources, with the addition of various levels of simulated noise. The total size of the DINGO cube is around 700GB.
This is where BOINC comes in. The huge cube is subdivded in to much smaller cubelets, around 10MB to 100MB each, that are processed by the computers of BOINC users. Each user runs one of the sourcefinding applications on their local machine, processes the small cubelet, then reports any sources found within the cubelet back to the project.
Once we’ve processed the full cube through the Duchamp and SoFiA sourcefinders, the results will be compared by a researcher, and a paper will be written on the comparitive performance of these sourcefinders.
We also have the prospect of processing cube data generated by ASKAP in the near future, as ASKAP has almost completed processing a large cube of sources.
The Sourcefinder project is run as part of the wider theSkyNet project.
We welcome any sort of feedback, advice, or bug reports. You can either make a post on the forums or send us an email at firstname.lastname@example.org . We're happy to hear from you.
SoFiA Beta Update 6
Apologies, I forgot to post my changelog for yesterday!
There was an issue with workunit name clashes between duchamp and sofia. The original naming convention was "runid_cubename", but there was no differentiation between SoFiA and Duchamp workunits.
The new naming convention is simply "appname_runid_cubename" to avoid the possibility of future clashes.
I've also contacted the SoFiA devs about the determinism issue I outlined last week and they're looking in to it.
I want to get started on real work as soon as possible, so if this week's work goes smoothly, I'll see about moving on to production work next week.
19 Oct 2017, 0:26:12 UTC · Discuss
SoFiA Beta Update 5
Reduced logging in the assimilator logs so I can actually find things in them.
Fixed an extremely unlikely bug in the assimilator that fires if amazon's S3 servers can't be reached.
Fixed a bug with the assimilator attempting to remove temporary folders that don't exist.
It also appears as though one of the SoFiA result parameters, Wm50, is non-deterministic. Some runs calculate it without error, but others fail. Here's an example error log from two different runs of the same parameter set:
Result 1 (courtesy of emoga):
Warning (Parametrization): Cannot determine kinematic major axis. Source too faint. Warning (Parametrization): Measurement of kinematic PA failed. Warning (Parametrization): Kinematic major axis derived from just 2 data points. Error (Parametrization): Calculation of Wm50 failed (3). Warning (Parametrization): Failed to measure source line width. Error (Parametrization): Cannot fit ellipse, source flux <= 0. Warning (Parametrization): Ellipse fit failed. Warning (Parametrization): Cannot determine kinematic major axis. Source too faint. Warning (Parametrization): Measurement of kinematic PA failed.Result 2 (courtesy of LCB001)
Warning (Parametrization): Cannot determine kinematic major axis. Source too faint. Warning (Parametrization): Measurement of kinematic PA failed. Warning (Parametrization): Kinematic major axis derived from just 2 data points. Error (Parametrization): Cannot fit ellipse, source flux <= 0. Warning (Parametrization): Ellipse fit failed. Warning (Parametrization): Cannot determine kinematic major axis. Source too faint. Warning (Parametrization): Measurement of kinematic PA failed.
SoFiA Beta Update 4
Fix yet another bug in the assimilator.
Modified the way SoFiA returns output files. There was an issue with various parameter combinations causing SoFiA to return outputs in different formats. This should be accounted for properly now.
Built a new, extended parameter set for SoFiA testing. We're now testing about 100 different combinations.
Pushed out new work units using the extended parameter set.
27 Sep 2017, 7:17:38 UTC · Discuss
SoFiA Beta Update 3
Fixed a few bugs in the assimilator.
Improved logging in the validator.
Out of the 1736 work units sent out in the last update, I have the results for 1723 of them stored in our cloud storage system. That's a 99% success rating!
There were a couple of issues with the assimilator that were preventing the results from being properly stored in our database though. We store sources found in our database as well as storing the raw files returned by a client in cloud storage.
Hopefully this will be the last week of the SoFiA beta, and then we can proceed in to full processing mode!
Edit: I've also updated the project description that appears on the main page to include SoFiA.
20 Sep 2017, 1:26:26 UTC · Discuss
SoFiA Beta Update 2
Increased VM RAM size to 256MB. This'll hopefully stop that furious swapping that was happening on Linux.
Compacted the VM image from 1.9GB down to 1.6GB. The compressed image is 446MB. I'll see if theres anything more I can strip out of the VM at a later date. I might be able to strip it down by another few hundred MB.
Released 3x the number of test work units as before. There should be around 1500 ready to go.
13 Sep 2017, 4:51:07 UTC · Discuss