inventory-provider merge requestshttps://gitlab.software.geant.org/geant-swd/dashboardv3/inventory-provider/-/merge_requests2024-03-15T09:55:41Zhttps://gitlab.software.geant.org/geant-swd/dashboardv3/inventory-provider/-/merge_requests/30Feature/pol1 775 poller interfaces cache built from preparsed2024-03-15T09:55:41ZSam RobertsFeature/pol1 775 poller interfaces cache built from preparsedChanges cache generation to use pre-parsed netconf caches in Redis. All unit tests pass as of writing.Changes cache generation to use pre-parsed netconf caches in Redis. All unit tests pass as of writing.Sam RobertsSam Robertshttps://gitlab.software.geant.org/geant-swd/dashboardv3/inventory-provider/-/merge_requests/29Improve testing speed2024-03-13T08:51:48ZPelle KosterImprove testing speedImproves testing speed by ~x3 (from ~3 min to ~1 min on my machine, parallel 12 threads):
* Load big 'router-info.json' only once, instead keep one non-mutating dict for all tests
* mock out ncc client to prevent trying to connect
* do ...Improves testing speed by ~x3 (from ~3 min to ~1 min on my machine, parallel 12 threads):
* Load big 'router-info.json' only once, instead keep one non-mutating dict for all tests
* mock out ncc client to prevent trying to connect
* do not setup logging in tests (mostly for not creating `error.log` and `info.log` files)
* some more minor changes...https://gitlab.software.geant.org/geant-swd/dashboardv3/inventory-provider/-/merge_requests/28[POL1-0703] add support for IC1 dashboard2024-03-12T09:15:41ZPelle Koster[POL1-0703] add support for IC1 dashboardhttps://gitlab.software.geant.org/geant-swd/dashboardv3/inventory-provider/-/merge_requests/27[DBOARD3-676]: Adding circuit types to the list of mic data2024-01-12T13:19:15ZSaket Agrahari[DBOARD3-676]: Adding circuit types to the list of mic datahttps://gitlab.software.geant.org/geant-swd/dashboardv3/inventory-provider/-/merge_requests/26Adding testing for mic2023-12-28T13:37:46ZSaket AgrahariAdding testing for michttps://gitlab.software.geant.org/geant-swd/dashboardv3/inventory-provider/-/merge_requests/25[DBOARD3-676]: Adding changes to add third party id support on inventory prov...2023-12-28T11:08:46ZSaket Agrahari[DBOARD3-676]: Adding changes to add third party id support on inventory provider sidehttps://gitlab.software.geant.org/geant-swd/dashboardv3/inventory-provider/-/merge_requests/24Feature/dboard3 833 handle inactive bundle config2023-11-20T11:01:47ZErik ReidFeature/dboard3 833 handle inactive bundle configDBOARD3-833DBOARD3-833Robert LattaRobert Lattahttps://gitlab.software.geant.org/geant-swd/dashboardv3/inventory-provider/-/merge_requests/23address issues with some speeds not being recognised, a timeout issue not...2023-11-20T11:47:00ZSam Robertsaddress issues with some speeds not being recognised, a timeout issue not...address issues with some speeds not being recognised, a timeout issue not being handled correctly and a bundle formatting erroraddress issues with some speeds not being recognised, a timeout issue not being handled correctly and a bundle formatting errorBjarke MadsenBjarke Madsenhttps://gitlab.software.geant.org/geant-swd/dashboardv3/inventory-provider/-/merge_requests/22Feature/pol1 744 fix speed issues2023-11-06T10:28:56ZSam RobertsFeature/pol1 744 fix speed issuesDoes not include IMS changes (need for this work still unclear), does fix the issues with the speed endpoint.Does not include IMS changes (need for this work still unclear), does fix the issues with the speed endpoint.Sam RobertsSam Robertshttps://gitlab.software.geant.org/geant-swd/dashboardv3/inventory-provider/-/merge_requests/21Draft: Feature/pol1 744 fix speed issues2023-10-20T14:38:55ZSam RobertsDraft: Feature/pol1 744 fix speed issues- Fix interface-info timeouts
- Make the poller speed endpoint use the data retrieved without it disappearing into the ether never to be used by anything- Fix interface-info timeouts
- Make the poller speed endpoint use the data retrieved without it disappearing into the ether never to be used by anythingSam RobertsSam Robertshttps://gitlab.software.geant.org/geant-swd/dashboardv3/inventory-provider/-/merge_requests/20Don't validate joblog:cached-response values2023-09-04T12:31:38ZBjarke MadsenDon't validate joblog:cached-response valuesWe constantly get jsonschema.ValidationError on the joblog:cached-response key, because it doesn't follow the structure of other joblogs:
```json
{"errors":[],"failed":[],"pending":[],"warnings":["'Returning cached interface info data f...We constantly get jsonschema.ValidationError on the joblog:cached-response key, because it doesn't follow the structure of other joblogs:
```json
{"errors":[],"failed":[],"pending":[],"warnings":["'Returning cached interface info data for mx1.gen.ch.geant.net'","'error loading interface-info data from mx1.gen.ch.geant.net'","'no cached interface info for intinfo:srx2.am.office.geant.net'","'error loading interface-info data from srx2.am.office.geant.net'","'error loading interface-info data from srx2.ch.office.geant.net'","'no cached interface info for intinfo:srx2.ch.office.geant.net'","'Returning cached interface info data for qfx.par.fr.geant.net'","'error loading interface-info data from qfx.par.fr.geant.net'","'error loading interface-info data from srx1.ch.office.geant.net'","'no cached interface info for intinfo:srx1.ch.office.geant.net'"]}
```Erik ReidErik Reidhttps://gitlab.software.geant.org/geant-swd/dashboardv3/inventory-provider/-/merge_requests/19Feature/pol1 689 router port speeds2023-08-15T13:38:04ZSam RobertsFeature/pol1 689 router port speedsSam RobertsSam Robertshttps://gitlab.software.geant.org/geant-swd/dashboardv3/inventory-provider/-/merge_requests/18Support ASN dot notation.2023-08-07T11:24:01ZNeda MoeiniSupport ASN dot notation.Neda MoeiniNeda Moeinihttps://gitlab.software.geant.org/geant-swd/dashboardv3/inventory-provider/-/merge_requests/17Draft: implement new port speeds2023-08-01T13:29:41ZBjarke MadsenDraft: implement new port speedshttps://gitlab.software.geant.org/geant-swd/dashboardv3/inventory-provider/-/merge_requests/16Parallel testing2023-07-28T08:37:21ZBjarke MadsenParallel testingThis merge request allows for tests to be run in parallel, resulting in about a 20x speedup of the test suite.
Locally, tests run to completion in under a minute, with the same coverage as shown in the previous release build process.
...This merge request allows for tests to be run in parallel, resulting in about a 20x speedup of the test suite.
Locally, tests run to completion in under a minute, with the same coverage as shown in the previous release build process.
My local coverage (my worker.py is slightly modified, ignore the difference in statements/coverage for that one):
```
---------- coverage: platform linux, python 3.8.10-final-0 -----------
Name Stmts Miss Cover
--------------------------------------------------------------------
inventory_provider/__init__.py 46 0 100%
inventory_provider/app.py 13 13 0%
inventory_provider/config.py 7 0 100%
inventory_provider/db/__init__.py 0 0 100%
inventory_provider/db/ims.py 188 35 81%
inventory_provider/db/ims_data.py 224 39 83%
inventory_provider/environment.py 25 1 96%
inventory_provider/juniper.py 168 15 91%
inventory_provider/routes/__init__.py 0 0 100%
inventory_provider/routes/classifier.py 482 87 82%
inventory_provider/routes/classifier_schema.py 13 0 100%
inventory_provider/routes/common.py 182 21 88%
inventory_provider/routes/data.py 53 1 98%
inventory_provider/routes/default.py 22 0 100%
inventory_provider/routes/jobs.py 82 6 93%
inventory_provider/routes/lg.py 42 2 95%
inventory_provider/routes/lnetd.py 55 2 96%
inventory_provider/routes/mic.py 16 1 94%
inventory_provider/routes/msr.py 471 58 88%
inventory_provider/routes/neteng.py 45 2 96%
inventory_provider/routes/poller.py 503 14 97%
inventory_provider/routes/testing.py 126 37 71%
inventory_provider/snmp.py 70 9 87%
inventory_provider/tasks/__init__.py 0 0 100%
inventory_provider/tasks/app.py 13 4 69%
inventory_provider/tasks/common.py 101 29 71%
inventory_provider/tasks/config.py 31 5 84%
inventory_provider/tasks/monitor.py 107 13 88%
inventory_provider/tasks/worker.py 792 189 76%
--------------------------------------------------------------------
TOTAL 3877 583 85%
========================================================================= 738 passed, 11 skipped, 11063 warnings in 55.51s =========================================================================
```
The coverage from the last release build:
```
========= 738 passed, 11 skipped, 10978 warnings in 1261.25s (0:21:01) =========
py38: commands[2]> coverage xml
Wrote XML report to coverage.xml
py38: commands[3]> coverage html
Wrote HTML report to htmlcov/index.html
py38: commands[4]> coverage report --fail-under 80
Name Stmts Miss Cover
--------------------------------------------------------------------
inventory_provider/__init__.py 46 0 100%
inventory_provider/app.py 13 13 0%
inventory_provider/config.py 7 0 100%
inventory_provider/db/__init__.py 0 0 100%
inventory_provider/db/ims.py 188 35 81%
inventory_provider/db/ims_data.py 224 39 83%
inventory_provider/environment.py 25 1 96%
inventory_provider/juniper.py 168 15 91%
inventory_provider/routes/__init__.py 0 0 100%
inventory_provider/routes/classifier.py 482 87 82%
inventory_provider/routes/classifier_schema.py 13 0 100%
inventory_provider/routes/common.py 182 21 88%
inventory_provider/routes/data.py 53 1 98%
inventory_provider/routes/default.py 22 0 100%
inventory_provider/routes/jobs.py 82 6 93%
inventory_provider/routes/lg.py 42 2 95%
inventory_provider/routes/lnetd.py 55 2 96%
inventory_provider/routes/mic.py 16 1 94%
inventory_provider/routes/msr.py 471 58 88%
inventory_provider/routes/neteng.py 45 2 96%
inventory_provider/routes/poller.py 503 14 97%
inventory_provider/routes/testing.py 126 37 71%
inventory_provider/snmp.py 70 9 87%
inventory_provider/tasks/__init__.py 0 0 100%
inventory_provider/tasks/app.py 13 4 69%
inventory_provider/tasks/common.py 101 29 71%
inventory_provider/tasks/config.py 31 5 84%
inventory_provider/tasks/monitor.py 107 13 88%
inventory_provider/tasks/worker.py 786 183 77%
--------------------------------------------------------------------
TOTAL 3871 577 85%
```Erik ReidErik Reidhttps://gitlab.software.geant.org/geant-swd/dashboardv3/inventory-provider/-/merge_requests/15feature/DBOARD3-771/include all sites in inventory provider /neteng/pops and ...2023-07-14T10:06:27ZMohammad Torkashvandfeature/DBOARD3-771/include all sites in inventory provider /neteng/pops and /neteng/pop/<lookup>Mohammad TorkashvandMohammad Torkashvandhttps://gitlab.software.geant.org/geant-swd/dashboardv3/inventory-provider/-/merge_requests/14only build snmp indices once per endpoint request2023-06-14T11:15:42ZSam Robertsonly build snmp indices once per endpoint requestSam RobertsSam Robertshttps://gitlab.software.geant.org/geant-swd/dashboardv3/inventory-provider/-/merge_requests/13Feature/dboard 743 build cache in worker process2023-06-14T10:42:26ZSam RobertsFeature/dboard 743 build cache in worker processSam RobertsSam Robertshttps://gitlab.software.geant.org/geant-swd/dashboardv3/inventory-provider/-/merge_requests/12Add redis auth credentials to celery backend config.2023-06-08T11:34:19ZNeda MoeiniAdd redis auth credentials to celery backend config.Neda MoeiniNeda Moeinihttps://gitlab.software.geant.org/geant-swd/dashboardv3/inventory-provider/-/merge_requests/11Added redis auth credentials to inventory config schema and updated _get_redi...2023-06-07T15:01:02ZNeda MoeiniAdded redis auth credentials to inventory config schema and updated _get_redis wrapper.Neda MoeiniNeda Moeini