> For the complete documentation index, see [llms.txt](https://monitoring.toolkit.citiobs.eu/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://monitoring.toolkit.citiobs.eu/data-integration/how-validate-raw-data.md). # How to validate raw data to create Analysis Ready Data? ## Description The Framework for Improving Low-cost Technology Effectiveness and Reliability (FILTER) provides advanced quality control and correction algorithms for air pollutants (fine particulate matter, PM2.5) and noise. The advantage of this tool is that it is sensing device agnostic, this is, it works for all type of devices, and it does not requiere that the device is co-located against a reference station. In its current version FILTER does requiere that the sensing device is nearby other sensing devices (low-cost sensor network, LCS). ## Why is this relevant? Harmonized, quality-controlled, and corrected datasets from various low-cost sensor networks are generally lacking. In order to increase the uptake of data from sensing devices, it is important that the user knows the quality of the data. Lack of data standardization and data of unknow quality limits the use of citizen generated data for official environmental assessments, policymaking, and scientific research. FILTER is a tool comprising a set of statistical algorithms for quality flagging and value correction. The FILTER tool is available on MATLAB and integrated as an open-source R package. The R package includes the full FILTER methodology along with detailed use cases to support applications to both historical and near real-time datasets. ## Useful resources Scientific paper describing the methodology: Hassani, A., Salamalikis, V., Schneider, P., Stebel, K., Castell, N. (2025).[ A scalable framework for harmonizing, standardization, and correcting crowd-sourced low-cost sensor PM2.5 data across Europe. Journal of Environmental Management](https://www.sciencedirect.com/science/article/pii/S030147972501076X), 380, 125100. Validated Dataset for Europe: ## You might also be interested in * [What is data quality? How can we increase data quality in citizen gathered data?](/managing-data/what-is-data-quality-how-can-we-increase-data-quality-in-citizen-gathered-data.md) * [What are the main aspects you need to consider when managing citizen collected data?](/managing-data/what-are-the-main-aspects-you-need-to-consider-when-managing-citizen-collected-data.md) --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter: ``` GET https://monitoring.toolkit.citiobs.eu/data-integration/how-validate-raw-data.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.