What is Sourcefinder?

Sourcefinder is a BOINC based distributed computing project that aims to provide a means to test the effectiveness of various astronomical sourcefinding applications. Sourcefinding applications are designed to process ‘cubes’ of data that contain radio sources and identify the location of the sources within the cube.
We’re currently testing the effectiveness of two different sourcefinding applications: Duchamp and SoFiA.

The effectiveness of a sourcefinding application can be determined by comparing the number of correct sources found within a cube to the number of incorrect sources found.
In order to provide a fair comparison between sourcefinding applications, we’re currently getting each application to process the same sections of a large cube known as the DINGO cube. DINGO is a simualated cube of data that contains a known catalogue of valid sources, with the addition of various levels of simulated noise. The total size of the DINGO cube is around 700GB.

This is where BOINC comes in. The huge cube is subdivded in to much smaller cubelets, around 10MB to 100MB each, that are processed by the computers of BOINC users. Each user runs one of the sourcefinding applications on their local machine, processes the small cubelet, then reports any sources found within the cubelet back to the project.
Once we’ve processed the full cube through the Duchamp and SoFiA sourcefinders, the results will be compared by a researcher, and a paper will be written on the comparitive performance of these sourcefinders.
We also have the prospect of processing cube data generated by ASKAP in the near future, as ASKAP has almost completed processing a large cube of sources.

The Sourcefinder project is run as part of the wider theSkyNet project.

We welcome any sort of feedback, advice, or bug reports. You can either make a post on the forums or send us an email at icrar.tsn.website@gmail.com . We're happy to hear from you.

Join Sourcefinder

User of the Day

User profile Profile shiva
Hello I'm Rick I live in western Kansas about 60 miles from the center of the USA. I'm a Crop Insurance Specialist. I work with farmers to protect...

News

New Sofia Beta App (Opt-In)
Hello everyone,

After the disaster that was last week, I took the advice of LumenDan and I've created a new app, sofiabeta. I'll be using this app in the future to trial new client versions and changes before using the main sofia app.

The sofiabeta app is opt in, so if you don't want to run it, you don't have to. You can opt into / out of running test applications here.

There's currently a set of test workunits waiting, which I've split into 6 runs to ease the ridiculous computation times.

I hope this goes smoothly.

Sam
31 Jan 2018, 4:58:04 UTC · Discuss


SoFiA Production Update 7 { Cancelled - See Forum for Details }
So the SoFiA devs managed to work out a fix to those extremely large memory requirements I mentioned last week, and I've been able to drop the VM memory requirements down to 2GB per VM. I know that's still a lot more than they were before, but it doesn't look like I'll be able to drop them any less without hitting some huge performance decreases (due to disk swapping).

The VM has been updated with this new version of SoFiA, and I've spend some extra time compacting it as far as possible. I managed to get the VM size below 500MB compressed!

The workunits from last week didn't release properly due to an issue with the workunit generator that I couldn't get around to fixing until now, but the test WUs should be flowing as I write. I'm interested to see how the system handles them.
24 Jan 2018, 5:09:09 UTC · Discuss


SoFiA Production Update 6
Hello everyone,

For this week, I'm planning on running a small set of 100MB workunits through SoFiA, just as a general test. There's only 705 workunits, but each workunit should take substantially longer to process than the ones we've been doing so far. For these test workunits, we're running all ~400 parameters on each workunit, rather than splitting the parameters up in to different runs. This is only going to be the case for the test workunits, the production ones will use split parameter sets just like the 10MB workunits did.

The SoFiA VM has been updated to use the latest version of SoFiA. Hopefully, this will help increase general stability and decrease some of the problems we've been encountering.

Each VM instance will now use up to 3072MB of memory, a huge increase from the previous 256MB. I'm sorry about this, but SoFiA doesn't seem to have particularly optimal memory usage, and my local tests were running into out of memory problems at anything less than 3GB of memory. You may want to limit the number of SoFiA instances your machines run if you begin to get performance issues. I've also reported the rather large memory usage to the developers, as it may be something they can improve on in future versions.

I'll be trying to visit the forums daily to keep an eye on any issues that might crop up, so hopefully I'll be able to resolve small problems in my spare time instead of having to postpone everything until next Wednesday.

I hope this goes smoothly.

Sam.
17 Jan 2018, 7:01:25 UTC · Discuss


Happy New Year 2018!
Hello everyone!

I'm back from my 2 week break, so work will be resuming later today once the final set of SoFiA workunits have been dispatched.
Once those final 15mb workunits have finished processing, we'll be moving on to the 100mb workunits.

Before that, though, I plan on updating the SoFiA VM to use the latest version of SoFiA (rather than the fairly old one we've been using) in the hopes that we'll see some speed and reliability improvements.

I'll run a few batches of test 100mb workunits, and we'll see how they go. If we don't have any significant issues, then we'll continue on and process the rest. If we run into issues, then we'll continue to run test batches until I can work everything out.

I hope you've all had an excellent holiday period too, and are looking forward to what this year brings. I certainly am.

Sam
10 Jan 2018, 1:01:56 UTC · Discuss


Merry Christmas / Happy Holidays 2017
Hello everyone!

I just want to wish you all a Merry Christmas, or Happy Holidays if you don't celebrate Christmas.
I hope everyone has an excellent time during this holiday period, because I know I certainly will.

I'm on break for the next 2 weeks, but I'll still be trying to release the remaining workunits for the current set, when I can find the time.
When I return, we'll get started crunching the larger 100mb workunits, which will hopefully see a more stable workunit supply rather than an on/off situation that we're used to.

Once again, Merry Christmas and a happy new year to everyone, and thanks for another year of Sourcefinder!

Sam.
25 Dec 2017, 8:51:20 UTC · Discuss


... more

News is available as an RSS feed   RSS


©2018 ICRAR