Infrastructure/arc
1
0
Fork 0
Code Issues 1 Pull requests Projects Releases Packages Wiki Activity Actions
275 commits 5 branches 0 tags 5 MiB
Commit graph

34 commits

Author SHA1 Message Date
siddharthvipul
f03df1e5f6 add pdc model 
Signed-off-by: siddharthvipul <siddharthvipul1@gmail.com>
 2021-04-29 15:29:11 +05:30
Pierre-Yves Chibon
2361325be6 Add a FAQ page for the monitoring search 
Signed-off-by: Pierre-Yves Chibon <pingou@pingoured.fr>
 2021-03-25 10:41:06 +01:00
Pierre-Yves Chibon
2149e8a446 Rework the stats script 
- Create a new object for each test-case/deployment
- Simplify the stats, no need to use defaultdict anymore
- Define the start_time to be the one of the test-case/deployment not
  the start time of the script (influences the requests per minute
  count)

Signed-off-by: Pierre-Yves Chibon <pingou@pingoured.fr>
 2021-02-16 23:20:34 +01:00
Pierre-Yves Chibon
dc8d7b0d87 Add adam's instance to the list of instances tested 
Signed-off-by: Pierre-Yves Chibon <pingou@pingoured.fr>
 2021-02-15 14:44:13 +01:00
Pierre-Yves Chibon
ef54186dbe Re-create the TestAPI object for each instance tested 
This ensure we have clean stats for each rather than keeping data from
the previous run into the next one.

Signed-off-by: Pierre-Yves Chibon <pingou@pingoured.fr>
 2021-02-15 14:42:15 +01:00
Pierre-Yves Chibon
019ae6b522 Adjust the msg_id for an older one to make sure it's present in all tests 
Signed-off-by: Pierre-Yves Chibon <pingou@pingoured.fr>
 2021-02-15 14:41:30 +01:00
Pierre-Yves Chibon
00a6aae964 Rename the constraint 
Signed-off-by: Pierre-Yves Chibon <pingou@pingoured.fr>
 2021-02-12 17:48:04 +01:00
Pierre-Yves Chibon
72ad2d72c7 Save some of the queries we've used for the record 
Signed-off-by: Pierre-Yves Chibon <pingou@pingoured.fr>
 2021-02-12 11:01:49 +01:00
Pierre-Yves Chibon
60425dbea1 Adding an index on the msg_id field of messages when it's partitioned 
Signed-off-by: Pierre-Yves Chibon <pingou@pingoured.fr>
 2021-02-11 15:26:05 +01:00
Pierre-Yves Chibon
0b38388b51 Add a unique constraint on msg_id/timestamp on messages2 
Signed-off-by: Pierre-Yves Chibon <pingou@pingoured.fr>
 2021-02-11 15:18:21 +01:00
Pierre-Yves Chibon
c69127dbe6 Adding an index on the id field of messages when it's partitioned 
Signed-off-by: Pierre-Yves Chibon <pingou@pingoured.fr>
 2021-02-11 14:15:54 +01:00
Pierre-Yves Chibon
2060e10a2b Add unique constraing and fix ownership 
Signed-off-by: Pierre-Yves Chibon <pingou@pingoured.fr>
 2021-02-11 12:39:44 +01:00
Pierre-Yves Chibon
7eb32d878a Update migration_timescaledb.sql as we drop foreign key constraints 
Signed-off-by: Pierre-Yves Chibon <pingou@pingoured.fr>
 2021-02-11 12:15:26 +01:00
Pierre-Yves Chibon
221b86edad Fix typo 
Signed-off-by: Pierre-Yves Chibon <pingou@pingoured.fr>
 2021-02-10 15:22:35 +01:00
Pierre-Yves Chibon
9971f132fb Restructure migration_timescaledb 
With this restructuration we adjust the FK before we make the messages2
table a hypertable.

Fingers crossed that'll help

Signed-off-by: Pierre-Yves Chibon <pingou@pingoured.fr>
 2021-02-09 17:20:54 +01:00
Pierre-Yves Chibon
c5ea7e4010 Some more changes/improvements to the migration_timescaledb.sql file 
Signed-off-by: Pierre-Yves Chibon <pingou@pingoured.fr>
 2021-02-09 11:48:36 +01:00
Pierre-Yves Chibon
b9ad3e9c75 Add logic to fix the foreign key 
Signed-off-by: Pierre-Yves Chibon <pingou@pingoured.fr>
 2021-02-09 11:47:11 +01:00
Pierre-Yves Chibon
1ff0d8fbd2 Improve logging and the get function and test our third instance 
Signed-off-by: Pierre-Yves Chibon <pingou@pingoured.fr>
 2021-02-05 16:54:54 +01:00
Pierre-Yves Chibon
346ff989e2 Fix foreign key constraint 
Signed-off-by: Pierre-Yves Chibon <pingou@pingoured.fr>
 2021-02-05 16:53:10 +01:00
Pierre-Yves Chibon
807b58df14 Drop statements that don't apply to timescaledb 
Signed-off-by: Pierre-Yves Chibon <pingou@pingoured.fr>
 2021-02-04 09:33:27 +01:00
Pierre-Yves Chibon
72b02b0ad8 Add a migration_timescaledb script 
This script contains the steps followed to install, configure,
enable, activate and convert the datanommer database to timescaledb

Signed-off-by: Pierre-Yves Chibon <pingou@pingoured.fr>
 2021-02-04 09:32:09 +01:00
Pierre-Yves Chibon
6b128c08b6 Add constraing on package_messages 
Signed-off-by: Pierre-Yves Chibon <pingou@pingoured.fr>
 2021-02-04 09:12:08 +01:00
Pierre-Yves Chibon
64fdf441d1 Add queries to fix the ownership and the foreign key on user_messages 
Signed-off-by: Pierre-Yves Chibon <pingou@pingoured.fr>
 2021-02-03 15:09:46 +01:00
Pierre-Yves Chibon
fd25669785 Adjust the migration.sql script to use range partition instead of list 
This allows to partition on the primary key and have all the partitions
be of the same size, instead of having them be by year with different
sizes.

Signed-off-by: Pierre-Yves Chibon <pingou@pingoured.fr>
 2021-02-01 14:54:51 +01:00
Will Woods
da5e593171 Fix SQL table constraints to work with partitioned table 
Partitioned table constraints like UNIQUE or PRIMARY KEY have to include
the table's partition key. This commit moves the constraints into the
CREATE TABLE statement and adds the partition key to those indexes.
 2021-01-26 13:21:50 -05:00
Pierre-Yves Chibon
9344d7eaf4 Add a couple of information/commands on the migration.sql 
Signed-off-by: Pierre-Yves Chibon <pingou@pingoured.fr>
 2021-01-26 15:27:31 +01:00
Pierre-Yves Chibon
a84d763068 Add a default table to the partition 
Signed-off-by: Pierre-Yves Chibon <pingou@pingoured.fr>
 2021-01-25 16:20:48 +01:00
Pierre-Yves Chibon
41f60da94f Drop un-used variable 
Signed-off-by: Pierre-Yves Chibon <pingou@pingoured.fr>
 2021-01-25 16:20:08 +01:00
Pierre-Yves Chibon
115d58d385 Don't assume all the messages have a year, at least for now 
Signed-off-by: Pierre-Yves Chibon <pingou@pingoured.fr>
 2021-01-25 15:14:47 +01:00
Pierre-Yves Chibon
eadb27c640 Adjust the how to save the year in the messages table 
Signed-off-by: Pierre-Yves Chibon <pingou@pingoured.fr>
 2021-01-25 15:14:00 +01:00
Pierre-Yves Chibon
b189c95395 Add first work on the migration SQL file to partition the DG DB 
Signed-off-by: Pierre-Yves Chibon <pingou@pingoured.fr>
 2021-01-25 15:13:10 +01:00
Pierre-Yves Chibon
c9f2e991b0 Improve the test_db_perfs script a bit 
* Reset the stats between runs so the number of fail/succeed runs
  show correctly
* Add a couple of URL to test
* Don't assume we always have a raw_messages in the JSON returned
* Print the time per requests even if there are no successfull requests

Signed-off-by: Pierre-Yves Chibon <pingou@pingoured.fr>
 2021-01-25 15:12:29 +01:00
Pierre-Yves Chibon
936f36ac3b Simplify the logging and just log to stderr for now 
Signed-off-by: Pierre-Yves Chibon <pingou@pingoured.fr>
 2021-01-22 13:15:05 +01:00
Pierre-Yves Chibon
cb3e97ad10 Add a script to test datagrepper's performances on different hosts 
Signed-off-by: Pierre-Yves Chibon <pingou@pingoured.fr>
 2021-01-22 11:28:56 +01:00