This project of the EFF on measuring the progress of AI Research (collectively elaborated through a notebook is hosted on Github) collects problems and metrics/datasets from the AI research literature, and tracks progress on them.
It is indeed a promising pilot to answer questions like:
- What is the ratio of hype to real progress in AI?
- What kinds of problems have been well solved by current machine learning techniques, which ones are close to being solved, and which ones remain exceptionally hard?
- And if the project is well followed up in the future : to what extent can such progress be used to foster social and economic development globally?
The pilot is using interesting methodologies that can help in understanding the current state of these technologies and raise meangful debate around the technical, political and legal issues and dilemmas they raise.
A promising start in defining metrics that could be enhanced with additional stakeholders’ perspectives including linkges to ‘AI for development’
Let us not forget that AI and machine learning methods can threaten safety of citizens particularly if in the hands of oppressive regimes which may raise wider and vital concerns than mere violation of privacy.