Tài liệu Data Management Issues Related to Drought Monitoring at Environment Canada pdf

10/24/2006
Data Management Issues Related to
Drought Monitoring at Environment
Canada
Robert Morris
Data Analysis and Archive Division
Meteorological Service of Canada
Toronto, Canada
Presentation at the North America Drought Workshop
Mexico City, October 19-20, 2006
10/24/2006
Outline
• Introduction
• Observational Networks and Data
Characteristics
• Quality Control
• Forward Looking – Data Management
Framework
• Summary
10/24/2006
Introduction
• The Canadian climatological archives contain temperature and
precipitation observations starting in 1840
1900 A.D. 159 stations 2000 A.D. 1313 stations
10/24/2006
Trends and Challenges
• Automatic observing technology introduces changes often faster
than data management keeps up resulting in
– dissimilar data sets available for analysis
– loss of long-term continuity (e.g. for comparison with long-term
normals)

• Rationalization of networks (e.g. reductions and operational control
reverting to other agencies such as observing sites at airports)
• Accessing and incorporating data from other agencies networks (e.g.
provinces, conservation authorities, electrical utilities)
• Increased requirement for accessing data in real-time
• Use of remotely sensed data (e.g. weather RADAR) and numerical
weather prediction model output to augment data sparse areas and
assist the quality control of in-situ data
10/24/2006
Environment Canada Daily T&P Data
• Synoptic data (e.g. data transmitted in WMO standard
synoptic formats on the GTS)
– Human observations – mainly airports
– Automatic stations
• Aviation
• Surface weather
• Reference Climate Stations (RCS)
• Cooperative / volunteer climate network
– Processed from paper forms
– Observer enters data on an Internet screen allowing for
more timely access if the observer is so inclined
Climate
Volunteers Contractors Autostations
Surface weather Aviation
Autostations Manned
National
Archive
System
(NAS)
DLY02

DLY44
DLY04
via
2304
paper
form
keypunch
via SM
via
2322
paper
form
keypunch
PRIM04
QC
RQC
via SM via SM
via
COOLTAP
…web
or IVR
Environment Canada Daily Climate Data Reporting and Processing
ACCESS
Available within hours or days Available within weeks or months
SMQC
A datamart with
access via web
services
is updated twice
daily with

options for each
station based on
prioity
DLY04, then
DLY02, then
DLY44
10/24/2006
Currently for September 2006
From
Synoptic
Reports
449 stations
From
volunteer
observers
(COOLTAP)
187 stations
Number
stations with
at least 28
observations
10/24/2006
Quality Control of Climate Data
• Based on “hard-wired” checks
– Physical and probable limits
– Physical consistency amongst elements
–Trends
– Spatial consistency
• Legacy system lacks flexibility to add new elements or
data streams or to incorporate new related data into

the system to assist checking e.g. RADAR, satellite,
NWP model output
• The Data Management Framework (DMF) will provide a
moderized approach for the future
Distribution
Station Maintenance Interface
Data Access
Policy/Maintenance Interface
Middleware: Common Data Access Routines
(Real-time, General Purpose)
Replicate
In
Replicate
out
Sites
Gateways
Servers
Modules for non Real-time
QA/QC (e.g. spatial, manual)
Data Acquisition
Decode and Transformations
Automatic (Real-time)
QA/QC
Alerting
Web Services
Data Access Policy
Station
metadata
Station
metadata

Short-Term Storage
Long-Term Storage
Original concept of the DMF component-based architecture*
User Applications
(e.g. Value Added)
Product Generators
Sites
G
a
t
e
w
a
y
s
Servers
Data
Management
Framework
QC Flag Summary Determination
QC Category 3
Physical Limits (Range)
QC Category 5
Temporal Consistency
QC Category 4
Inter-Variable Consistency
QC Category 6
Spatial Consistency
QC Functions
Category 1

Flag
Category 3
Summary Flag
Category 4
Summary Flag
Category 5
Summary Flag
QC
Summary
Flag
Outputs
QC
Flag
Basic Test
Default tests
Test
Test …
Default tests
Test
Test …
QC
Flag
QC
Flag
Default tests
Test
Test …
QC
Flag
QC

Flag
Default tests
Test
Test …
QC
Flag
QC
Flag
QC Category 1
Data Element Presence
Category 6
Summary Flag
Interact with other data
elements and QC flags
QC Category n
QC
Flag
QC
Flag
QC Category 2
Data Element Integrity
QC
Flag
Default tests
Test
Category 2
Flag
QC Category
Data Element
QC

Flag
QC
Flag
QC
Flag
QC
Flag
QC
Message
QC
Message
QC
Message
QC
Message
QC
Message
QC
Message
QC
Message
QC
Message
QC
Message
QC
Message
QC
Message
QC

Message
Data/Metadata Out
QC in the DMF
10/24/2006
Summary
• Climate data for drought monitoring in Environment
Canada is steadily improving, especially in the area of
availability of near-real time data.
• Improvements in data management are planned in the
areas of
– Near-real time quality control
– Incorporating new data elements and other agencies’
data
– Use of related data (e.g. RADAR) to assist quality control
and data completeness
– Metadata, access systems (e.g. web services),
geomatics systems
10/24/2006
Gracias por su atención
Thank you