Seedme2 is building modular web-based building blocks to enable computational science teams to quickly share, visualize, and discuss the results of simulations and analyses run using the latest supercomputing resources.
The increasing availability of High Performance Computing (HPC), cloud computing, and high-resolution high-update rate sensing and imaging instruments has enabled researchers to model, observe, and analyze an ever-widening range of scientific phenomena. Many computing tasks are highly iterative in nature, as the right input parameters are rarely known ahead of time. This leads to a repeating process – run a job, assess results, refine parameters, run another job, and so on. Performing this process well requires rapid access and assessment of transient data and preliminary results. Does the output look reasonable? Did computation converge in an expected way? Did the job use the compute power efficiently or did it bottleneck somewhere? With timely feedback, an erroneous job can be terminated before it wastes precious compute and researcher time. With rapid access to transient data and brief visualizations, job parameters can be refined quickly and computation performed again. Unfortunately, swift informative job feedback and light-weight visualizations difficult to engineer within an HPC environment.
If you could do big computation on a local desktop computer, your applications could provide simple progress bars, scrolling status messages, and quick progress visualizations of preliminary results saved to local disks. But HPC jobs don’t work this way. They're run on big remote machines that don't have displays or direct interactive connections back to researchers. They don't have progress bars, dialogs, windows of status messages, or any way to quickly visualize data and show it to researchers and their colleagues. Instead, a computational job is submitted to a batch queue to be run at an unknown time in the future. When it runs, its results are typically directed to output files that remain unavailable until the job completes. Current computation cyberinfrastructure has poor feedback mechanisms.
On successful job completion, output data needs to be processed to extract meaningful preliminary results. Generating these results may requires post processing to build quick visualizations, images, and animation videos. This delays submission of the next job iteration with improved parameters. And when these preliminary results are generated, sharing them among collaborators is often limited to emails back and forth, which can’t easily include all of the relevant data and appropriate visual ways to display it. Current cyberinfrastructure has limited support for sharing preliminary results.
This project is addressing cyberinfrastructure weaknesses by creating new building blocks to support rapid web-based data sharing, granular access controls, light-weight microformats for scientific data, and brief web-based visualizations to enable quick looks at transient data and preliminary results from HPC jobs. These building blocks are cast as general-purpose APIs and Drupal-specific modules that may be used to quickly stand up web-based project-, lab-, and department-based data sharing services.
This project is addressing cyberinfrastructure weaknesses by creating new building blocks to support rapid web-based data sharing, group-based access controls, light-weight microformats for scientific data, and brief web-based visualizations to enable quick looks at transient data and preliminary results from HPC jobs. These building blocks are cast as general-purpose APIs and Drupal-specific modules that may be used to quickly stand up web-based project-, lab-, and department-based data sharing services.
- Amit Chourasia, Principal Investigator
- Michael Norman, Co-Principal Investigator
- David Nadeau, Technical Architect
- Andrew Ferbert, Systems Administrator
- Michael Dwyer, Database Administrator
This work is supported by the National Science Foundation under Grant No. 1443083. "Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation."