Latest revision as of 09:14, 24 January 2015

U.S. national oil and gas data

U.S. Energy Information Administration (EIA)

Archives of past Annual Energy Outlooks (aka AEOs—both the pdf reports and the data behind them); Monthly Energy Review (which includes US & OECD petroleum stocks); U.S. petroleum stocks (broken down by crude oil and various petroleum products); explanation of natural gas spot price data. AEO Working Group presentation slides and summaries of meetings. Working Paper Series. International Energy Statistics, including monthly oil production by country, and by type (crude + condensate, natural gas plant liquids, other & refinery gain).

Natural gas production: monthly and annual (separated into oil wells, gas wells, shale wells, coalbed methane). Also coalbed methane by state.

Other sources

FracFocus: voluntary disclosures of chemicals used in fracking.

Fractracker: a huge amount of data and maps, including: list of U.S. oil and gas wells (except for Texas).

U.S. state oil and gas data

Texas

Online Research Queries (old) and Oil & Gas Data Query (new)—particularly useful are the Production Data Query (old version without downloads or new version with downloads) , and Drilling Permit (W-1) Query. Monthly updates on crude oil, natural gas, etc (replacement for the old “OGISMCON” pdf reports). Data sets with lat & long available to order. New GIS viewer.

North Dakota (free)

ArcIMS interactive map of wells (shows rigs, wells, horizontal legs), individual well search (gives status, but not production stats), list of daily permits (such as this from Feb 4, 2013), FAQ, list of county maps. Confidential well list (most wells in the state are put on confidential status for the first six months or so after they’re permitted). General data page on Bakken-Three Forks. Bakken Horizontal Wells By Producing Zone, showing cumulative production (the zones are: Middle Bakken, Three Forks, Middle Bakken/Three Forks, Lodgepole, Upper Bakken, Lodgepole/Middle Bakken). University of North Dakota’s EERC map of Bakken drilling (with lateral legs, well quality, separation by bench and by sub-play, and more). ND monthly production by county (link to pdf), which excludes wells on “confidential” status, which is most Bakken wells during their first 6 months; the Monthly Production Report Index gives estimates of the monthly production by county that include confidential wells. ND active rig list.

North Dakota (paid)

Basic subscription ($50/year): gives access to monthly production data for every well (including oil, gas sold, gas flared or vented, and water produced). Also access to GIS system that shows each well.

Ohio

Department of Natural Resources: Oil and Gas Well Production files, including xls files for: Utica shale (annual for 2011, 2012), horizontal shale wells (quarterly from start of 2013). Risk Based Data Management System (RBDMS) [requires MS Access; used by Fractracker]. Interactive map of oil and gas wells.

Louisiana

Department of Natural Resources: SONRIS (oil & gas database); main Facts & Figures page, natural gas production (wet gas after lease separation); historical rig counts by region & compared with all US (plus a summary comparison), and a weekly rig count update. Graph of the rig counts in Haynesville play from the site Haynesvilleplay.com (still being updated as of Oct 2014).

Montana

monthly production: total and by field.

West Virginia

oil and gas production data; oil and gas well search.

Pennsylvania

Department of Environmental Protection site on oil and gas activity: production reports, statewide data downloads, and a list of various other “reports” (forms for requesting large lists of data). Direct links: list of all wells spud, sums of wells spud by operator, list of individual permits (inc lat & long). Pennsylvania Geological Survey: maps and digital data.

Arkansas

Arkansas Geological Survey page on Fayetteville shale has: list of wells (with annual production totals for individual wells) & county maps showing wells. Arkansas Oil and Gas Commission: Online Production and Well Database, an old-school Java application (on my Mac, I had to go into System preferences, Java, Java Control Panel, Security, and under “exception site list” add two trusted domains: http://aogc.state.ar.us http://aogc.state.ar.us & http://170.94.97.201. In my browser (Firefox) I also allows these domains to create pop-up windows, and to install add-ons.

Colorado

COGIS , which has a Production Data Inquiry

Alaska

Revenue Sources Book; Department of Natural Resources.

California

Department of Conservation, Division of Oil, Gas & Geothermal Resources (DOGGR) has data on wells, including stimulation techniques. South Coast Air Quality Management District (SCAQMD) has data on chemical releases.

Miscellaneous oil & gas data sources

Baker-Hughes:

Rig Count Overview & Summary (U.S. and international), Rig Counts Report (inc by state, by play), Well Count Report (inc by play).

Multi-state Shale Research Collaborative:

Misc data on shale plays

Miscellaneous mapping data and tools

US Census Bureau TIGER Products (kml files and more, for county outlines and the like), FIPS codes for each state.

Fractracker: a huge amount of data and maps, including: list of U.S. oil and gas wells (does not include Texas).

Tilemill: fantastic open-source, free mapping software.

US Geological Survey: energy maps (wind, geothermal, oil & gas, more); Energy Data Finder (inc. boundaries for oil and gas provinces).

U.S. Energy Information Administration (EIA): maps and shapefiles, including for pipelines.

Vessel Finder: track the movements of ships, including oil tankers.

National and international energy data

U.S. National Renewable Energy Laboratory (NREL):

Energy Data Exchange

U.S. Bureau of Safety and Environmental Enforcement (BSEE):

offshore oil Production Data Online Query

U.S. Bureau of Labor Statistics:

data homepage (includes electricity prices)

Miscellaneous U.S. federal data:

Map of nuclear waste disposal sites (Wikimedia, from DOE; version on Archive.org)

International Energy Agency (IEA):

statistics

HYDE database:

Long-term world energy data from the Netherlands Environmental Assessment Agency.

OPEC:

Monthly Oil Report: “covers major issues affecting the world oil market and provides an outlook for crude oil market developments for the coming year. The report provides a detailed analysis of key developments impacting oil market trends in world oil demand, supply as well as the oil market balance.”

BP Statistical Review:

Main page, with data on oil, gas, solar, wind, and more. Direct link to 2014 data sheets (xlsx file). Direct links to 2013 version (xlsx file).

WARNING: BP’s widely cited numbers for “proved reserves” of oil can’t be taken at face value for many nations, since it is in many ways apples-to-oranges comparison across nations. BP puts a disclaimer on these numbers, but it is widely ignored. That disclaimer:

“The data series for proved oil and gas reserves in BP Statistical Review of World Energy June 2014 does not necessarily meet the definitions, guidelines and practices used for determining proved reserves at company level, for instance, under UK accounting rules … or as published by the US Securities and Exchange Commission, nor does it necessarily represent BP’s view of proved reserves by country.” Gas in Focus (Observatoire du Gaz):

Data downloads on gas storage, LNG import facilities, and more.

World Bank:

monthly commodity prices 1960-present, including oil and natural gas (aka “Pink Sheet” or “Pink Data”)

Russia:

RIATEK (has English pages); CDU TEK (by purchase only?)

Japan:

Ministry of Economy, Trade, and Industry: stats page (in English)

China:

China Geological Survey

Iraq:

oil exports: http://somooil.gov.iq/en/

Norway:

Norwegian Petroleum Directorate; Statbank (aka Statistikksbanken) with stats on Norway’s oil investments

Renewable Energy Policy Network for the 21st Century (REN21): reports with annual data on various renewable energy sources, by country

Germany:

Ministry of Energy: energy data (to go along with big 2014 report)

Nigeria:

National Petroleum Corporation oil and gas statistics

IMF commodities: natural gas, oil, etc: http://www.imf.org/external/np/res/commod/index.aspx

Wall Street Journal prices: http://online.wsj.com/mdc/public/page/2_3023-cashprices.html

Transco Zone 6 natural gas spot prices (and more): http://www.quandl.com/WSJ/NG_TRAN6-Natural-Gas-Transco-Zone-6-NY

Indexmundi: a compendium of many sources of data

Web scraping tools & tips:

Anaconda: a “wrapper” for Python, which makes it very easy to run an iPython notebook on your computer.

Scrapy: a Python-based web scraping library

The Bastard’s Book of Regular Expressions (a free e-book): Web designers often use regular expressions to add styles to their pages, but they can also be used to reverse engineer the page, in a sense, to select particular parts of the pages so that you can scrape their content.

@@ Line 15: / Line 15: @@
 = U.S. state oil and gas data =
-=== Texas: ===
+=== Texas ===
 [http://www.rrc.state.tx.us/data/online/ Online Research Queries] (old) and [http://webapps2.rrc.state.tx.us/EWA/ewaMain.do Oil & Gas Data Query] (new)—particularly useful are the Production Data Query ([http://webapps.rrc.state.tx.us/PDQ/home.do old version without downloads] or [http://webapps.rrc.state.tx.us/PDQ/home.do new version with downloads]) , and [http://webapps2.rrc.state.tx.us/EWA/drillingPermitsQueryAction.do Drilling Permit (W-1) Query]. [http://www.rrc.state.tx.us/oil-gas/research-and-statistics/production-data/texas-monthly-oil-gas-production/ Monthly updates] on crude oil, natural gas, etc (replacement for the old “OGISMCON” pdf reports). Data sets with [http://www.rrc.state.tx.us/about-us/resource-center/research/data-sets-available-for-purchase/drilling-permit-data/ lat & long] available to order. [http://wwwgisp.rrc.state.tx.us/GISViewer2/ New GIS viewer].
-=== North Dakota (free): ===
+=== North Dakota (free) ===
 [https://www.dmr.nd.gov/OaGIMS/viewer.htm ArcIMS interactive map] of wells (shows rigs, wells, horizontal legs), [https://www.dmr.nd.gov/oilgas/findwellsvw.asp individual well search] (gives status, but not production stats), list of daily permits (such as this from [https://www.dmr.nd.gov/oilgas/daily/2013/dr020413.pdf Feb 4, 2013]), [https://www.dmr.nd.gov/oilgas/webhelpfaq.asp FAQ], list of [http://www.dot.nd.gov/road-map/county-base/ county maps]. [https://www.readability.com/reading-list Confidential well list] (most wells in the state are put on confidential status for the first six months or so after they’re permitted). [https://www.dmr.nd.gov/ndgs/bakken/bakkenthree.asp General data page] on Bakken-Three Forks. [https://www.dmr.nd.gov/oilgas/bakkenwells.asp Bakken Horizontal Wells By Producing Zone], showing cumulative production (the zones are: Middle Bakken, Three Forks, Middle Bakken/Three Forks, Lodgepole, Upper Bakken, Lodgepole/Middle Bakken). University of North Dakota’s EERC [https://www.beaconreader.com/mason-inman/map%20of%20Bakken%20drilling map of Bakken drilling] (with lateral legs, well quality, separation by bench and by sub-play, and more). ND monthly [https://www.dmr.nd.gov/oilgas/stats/countymot.pdf production by county] (link to pdf), which excludes wells on “confidential” status, which is most Bakken wells during their first 6 months; the [https://www.dmr.nd.gov/oilgas/mprindex.asp Monthly Production Report Index] gives estimates of the monthly production by county that include confidential wells. ND [https://www.dmr.nd.gov/oilgas/riglist.asp active rig list].
-=== North Dakota (paid): ===
+=== North Dakota (paid) ===
 [https://www.dmr.nd.gov/oilgas/basicservice.asp Basic subscription] ($50/year): gives access to monthly production data for every well (including oil, gas sold, gas flared or vented, and water produced). Also access to GIS system that shows each well.
-=== Ohio: ===
+=== Ohio ===
 Department of Natural Resources: [http://oilandgas.ohiodnr.gov/production Oil and Gas Well Production files], including xls files for: Utica shale (annual for 2011, 2012), horizontal shale wells (quarterly from start of 2013). [http://oilandgas.ohiodnr.gov/industry/rbdms Risk Based Data Management System] (RBDMS) [requires MS Access; used by Fractracker]. [http://oilandgas.ohiodnr.gov/well-information/oil-gas-well-locator Interactive map] of oil and gas wells.
-=== Louisiana: ===
+=== Louisiana ===
 Department of Natural Resources: [http://sonris.com/ SONRIS] (oil & gas database); main [http://dnr.louisiana.gov/index.cfm?md=pagebuilder&tmp=home&pid=208 Facts & Figures] page, natural gas production (wet gas after lease separation); [http://dnr.louisiana.gov/assets/TAD/data/drill_weekly/ogj_rig_count.pdf historical rig counts] by region & compared with all US (plus a [http://dnr.louisiana.gov/assets/TAD/data/drill_weekly/ogj_rig_comp.pdf summary comparison]), and a [http://dnr.louisiana.gov/assets/TAD/data/drill_weekly/WeeklyRigCountUpdate.pdf weekly rig count update]. Graph of the [http://www.haynesvilleplay.com/ rig counts in Haynesville play] from the site Haynesvilleplay.com (still being updated as of Oct 2014).
-=== Montana: ===
+=== Montana ===
 monthly production: [http://www.bogc.dnrc.mt.gov/webapps/dataminer/Production/ProdMonthlyTotal.aspx total] and [http://www.bogc.dnrc.mt.gov/webapps/dataminer/Production/ProductionMonthlyByField.aspx by field].
-=== West Virginia: ===
+=== West Virginia ===
 oil and gas [http://www.dep.wv.gov/oil-and-gas/databaseinfo/Pages/default.aspx production data]; oil and gas [https://apps.dep.wv.gov/oog/wellsearch_new.cfm well search].
-=== Pennsylvania: ===
+=== Pennsylvania ===
 Department of Environmental Protection site on [https://www.paoilandgasreporting.state.pa.us/publicreports/Modules/Welcome/Welcome.aspx oil and gas activity]: [https://www.paoilandgasreporting.state.pa.us/publicreports/Modules/Production/ProductionHome.aspx production reports], [https://www.paoilandgasreporting.state.pa.us/publicreports/Modules/DataExports/DataExports.aspx statewide data downloads], and a list of various other “[http://www.portal.state.pa.us/portal/server.pt/community/oil_and_gas_reports/20297 reports]” (forms for requesting large lists of data). Direct links: [http://www.depreportingservices.state.pa.us/ReportServer/Pages/ReportViewer.aspx?/Oil_Gas/Spud_External_Data list of all wells spud], [http://www.depreportingservices.state.pa.us/ReportServer/Pages/ReportViewer.aspx?/Oil_Gas/Wells_Drilled_By_Operator sums of wells spud by operator], [http://www.depreportingservices.state.pa.us/ReportServer/Pages/ReportViewer.aspx?/Oil_Gas/Permits_Issued_Detail list of individual permits] (inc lat & long). Pennsylvania Geological Survey: [http://www.dcnr.state.pa.us/topogeo/econresource/oilandgas/marcellus/marcellus_maps/index.htm maps and digital data].
-===Arkansas: ===
+===Arkansas ===
 ''Arkansas Geological Survey'' [http://www.geology.ar.gov/fossilfuel_maps/fayetteville_play.htm page on Fayetteville shale] has: list of wells (with annual production totals for individual wells) &amp; county maps showing wells. ''[http://www.aogc.state.ar.us Arkansas Oil and Gas Commission]'': [http://www.aogc.state.ar.us/JDesignerPro/JDPArkansas/default.htm Online Production and Well Database], an old-school Java application (on my Mac, I had to go into System preferences, Java, Java Control Panel, Security, and under “exception site list” add two trusted domains: http://aogc.state.ar.us http://aogc.state.ar.us &amp; http://170.94.97.201. In my browser (Firefox) I also allows these domains to create pop-up windows, and to install add-ons.
-=== Colorado: ===
+=== Colorado ===
 [http://cogcc.state.co.us COGIS] , which has a  Production Data Inquiry
-=== Alaska: ===
+=== Alaska ===
 [http://www.tax.alaska.gov/programs/sourcebook/index.aspx Revenue Sources Book]; [http://dnr.alaska.gov Department of Natural Resources].
-=== California: ===
+=== California ===
 Department of Conservation, [http://www.conservation.ca.gov/dog/Pages/Index.aspx Division of Oil, Gas &amp; Geothermal Resources] (DOGGR) has data on wells, including stimulation techniques. [http://www.aqmd.gov South Coast Air Quality Management District] (SCAQMD) has data on chemical releases.
-=== Other oil & gas data sources ===
+= Miscellaneous oil & gas data sources =
-Baker-Hughes: Rig Count Overview & Summary (U.S. and international), Rig Counts Report (inc by state, by play), Well Count Report (inc by play).
+=== Baker-Hughes: ===
+[http://phx.corporate-ir.net/phoenix.zhtml?c=79687&amp;p=irol-rigcountsoverview Rig Count Overview &amp; Summary] (U.S. and international), [http://gis.bakerhughesdirect.com/Reports/RigCountsReport.aspx Rig Counts Report] (inc by state, by play), [http://phx.corporate-ir.net/phoenix.zhtml?c=79687&amp;p=irol-wellcountus Well Count Report] (inc by play).
-Multi-state Shale Research Collaborative: Misc data on shale plays
+=== Multi-state Shale Research Collaborative: ===
+[http://www.multistateshale.org/shale-resources Misc data on shale plays]
-Mapping data
-US Census Bureau TIGER Products (kml files and more, for county outlines and the like), FIPS codes for each state.
+=== Miscellaneous mapping data and tools ===
-Fractracker: a huge amount of data and maps, including: list of U.S. oil and gas wells (does not include Texas).
+'''US Census Bureau''' [https://www.census.gov/geo/maps-data/data/tiger.html TIGER Products] (kml files and more, for county outlines and the like), [http://www.epa.gov/envirofw/html/codes/state.html FIPS codes] for each state.
-Tilemill: fantastic open-source, free mapping software.
+'''[http://fractracker.org/ Fractracker]''': a huge amount of data and maps, including: [http://www.fractracker.org/downloads/us-oil-and-gas-wells-february-2014 list of U.S. oil and gas wells] (does not include Texas).
+'''[https://www.mapbox.com/tilemill Tilemill]''': fantastic open-source, free mapping software.
 US Geological Survey: energy maps (wind, geothermal, oil & gas, more); Energy Data Finder (inc. boundaries for oil and gas provinces).

Track changes made to this page

Difference between revisions of "Mason Inman's data sources"