For big data problems, you need a big solution. Fortunately, this solution does not have to be restricted to an ArcGIS workspace.
Meet Marco Commander, the server-side component to Marco Desktop built to assist your team in Spatial Data Discovery and Data Reference Inventory. It is designed for parallel processing and server-class machines with the sole purpose of being the best Knowledge Management system that your network has ever seen.
IN ONE PLACE
Because Marco Commander gathers detailed synopses of spatial data, it must include a way to store this information with immediate and secure access. Merging these ideas, the Marco Database was born.
As tools are run within Marco Commander, results are stored in a database made up of nearly twenty tables. Supported databases included SQLite for quick access as well as SQL Server and Oracle for more secure platforms.
With this approach, Database Administrators may open the Marco Database at any point to view results or define locations for additional scans. This resource can then be uploaded in Marco Mystic for full search, filter, and analysis capabilities to better understand the data at hand.
ON A GRAND SCALE
With much of Marco Commander's functionality geared towards Spatial Data Discovery and Spatial Data Inventory, the tools included in this application allow spatial data details to be gathered that are not likely acquired by common Knowledge Management software.
To name a few, these attributes include:
Full Path to ArcGIS Map Document, Layer File, File Geodatabase, Folder, ArcSDE, etc.
Date of Creation, Modification, and Last Access
Status (i.e., Moved, Deleted, Crashed, Unknown)
Spatial Reference and Extents of Dataset
Spatial Reference and Extents of Bookmarks and Data Frames
The data consumed by Marco Commander does not always have to begin as geospatial. Many non-spatial files, like Microsoft Word (.doc) and Portable Document File (.pdf), contain references to geographic locations in their content that can be geoenriched, turning these once simple documents into points on a map.
BREAK OUT SPATIAL DATA
A benefit of Knowledge Management software is that it permits not only the discovery of data, but also the opportunity to organize it into more manageable and meaningful resources. One approach to wrangling in this information is the categorization of spatial data, where custom categories may be applied to Spatial Data Discovery results to learn which files match specific terms and keywords.
These matches are output in the form of a series of reports, making it even easier to implement their care in your own management practices.
RECORD & MAP COORDINATES
The Spatial Data Discovery tools within Marco Commander pull a variety of information from each piece of data it encounters. One such detail is that of extent coordinates for datasets, bookmarks, and data frames alike.
Once recorded in the Marco Database, these can then be visualized within ArcGIS to help determine any datasets not matching current standards - or simply as a matter of interest.
REPORT ON USER
ACTIVITY & OWNERSHIP
While a Knowledge Management system can assist organizations in getting a better grasp on what data it is they have at their disposal, true management of these valuable assets are the users that maintain the quality of each dataset.
Marco Commander allows Managers and Team Leads insights into how well their teams are accomplishing this by creating reports based on the number of files as well as their health, status, and location credited to each unique user found.
REPOINT BROKEN FILE PATHS
With all of that data, there may be any number of ArcGIS Map Documents containing broken data sources. Since Marco Commander is run via Command Prompt, resetting these paths inside of a physical ArcMap workspace is not the most effective use of our time.
Instead, the application employs the same Remapping functionality found in Integrated Marco Desktop that provides users with the opportunity to define data reference connections for all broken files in a central location, known as a Remap File, followed by the automated repointing process against these ArcGIS Map Documents and datasets.
REPORT ON DATA & STATISTICS
The reporting capabilities of Marco Commander do not end at compiling insights on its users. Its diverse range of tools allow your team to generate reports detailing the number of ArcGIS Map Documents, total layers, broken layers, feature classes, raster datasets, raster catalogs, and mosaic datasets found at each container and folder connection defined during the Spatial Data Discovery process.
METADATA AT ITS MOST MINUTE
The Integrated Marco Studio suite is essential in helping teams develop and adhere their own data standards. Many organizations ensure distinct metadata elements be attributed to the datasets it owns. The question is, how do we keep track of this when we are responsible for so many files?
By including metadata in those attributes pulled during the Spatial Data Discovery process, we save time rather than reviewing this information by opening each dataset individually. These details can then be more closely inspected to determine how in sync their contents are with preferred Knowledge Management standards.
RECORD HISTORICAL ATTRIBUTES
Is your organization undergoing a data overhaul? Are you making strides to clean-up outdated and neglected files in an effort to return your system to its once spotless condition?
We know how reassuring it is to be able to see what progress has been made - even if we would rather not dwell in the past. Tools within Marco Commander are designed to store details about each inventory process in a separate location within the Marco Database to serve as a historical marker. Only the most pertinent information, like identification, location, and health, are saved in this way. However, this is essential should you need to see just how far you have come.
DUST OFF THE DATABASE
The Marco Database is responsible for many an important detail it inventories. As it turns out, it also keeps track of those it cannot inventory.
When an error is encountered for an ArcGIS Map Document or dataset, it is recorded in the database. These errors are typically consequences of corrupt documents or files that do not open within the time allotted. Rather than halting processing completely to detail each indiscretion, recording them within the Marco Database allows you to address them in your own time.
While this is helpful, we realize that you do not always need these details as much as you need...say, to know what did work. For this purpose, Marco Commander also includes tools to perform its own bit of maintenance on the Marco Database by cleaning up error tables on command.
MULTIPROCESSING & MULTITHREADING
We know more is better, and we also know that Marco Commander would have to agree.
Server-based applications have a unique opportunity not always available to their Desktop counterparts. That is, the chance to do more faster. Marco Commander embraces this to its fullest, with many of its features employing multiprocessing across servers. The degree to which this is done is fully customizable by the user to fit the network, available time, and amount of data being crawled.