nosql - Tuning write performance in cassandra -


We have a typical scenario:

10 simple columns less than 10 column families

When we receive a request from the client, we need to write 10 000 000 records of this column family in the database and we are writing them in batch (1000 in a batch) It usually lasts 5-10 minutes depending on the number of nodes in cluster and replication factor.

After writing in the next few hours, we will get many updates (each record is updated 2 times).

So many of us write / update at a time (one hour) of the day and after that are very few.

Question: For the purpose of improving the write / update performance, I have looked at the memtable_flush_queue_size and similar configuration fields for example I do not have enough experience with the cassandra, to know what to do.

Any suggestion is helpful,

Evan

  1. Increase JVM Memory (maximum 12 GB on Java 6+) - This will automatically increase the size of memtables and reduce the flush interval, which also means that will be merged simultaneously in continuously updated RAM and coincidentally Not during - this will also reduce the disk usage
  2. Very important: Use separate disk for data and use to log in Switch to. You can use SSD for data, there is no substance to logging, since this sequential writes
  3. Changing the replication factor to 1 will generate less weight in the cluster, because each node has its The data should be kept in mind and there are no replicas, but you can lose data - I will not recommend it again.

    This can help to get some better understanding:



Comments

Post a Comment

Popular posts from this blog

python - how we can use ajax() in views.py in django? -

I have a little code here. Could you please tell me what this code really does? please. Here in the AJAX call: $ .ajax ({url: "{% url} upload_document '%}", type: "POST", data : {Csrfmiddletoken: document.getElementsByName ('csrfmiddletoken') [0] Price, Title: document.getElementById ('title'). Value, // document: document: document.getElementById ('document'),}, datatype: "Jason", success: function (feedback) {if (feedback == "true") {// success} other {// attachment errors}}}); How AJAX works in the Django and how we can see the AJAX request. Here is the idea of ​​my thoughts def upload_document (requested): Print request. POTT print request FILES if request.is_ajax (): If request.method == 'POST': form = UploadForm (request.POST, request.FILES, user = request.user) if form.is_valid (): Form.save () return HTTPPCS (SimpleJason Dumps ('True'), Mime Type = 'App / Jason') Other:...
Read more

matlab - Using loops to get multiple values into a cell -

I have 31 topics (S1, S2, S3, S4, etc.) 3 images in each topic, opposite 1 IMG, Contrast 2 IMG and Contrast 3 Are IMG I p In a NX1 cell named P, I would like to use a loop to get all the paths of contrasts from all the subjects: data / S1 / contrast1.img / S1 / contrast2.img data / S1 / contrast3.img data / s2 / contrast1.img data / S2 / contrast2.img Data / S2 / contrast3.img ... Data / S31 / contast3.img This is what I've tried: A = {'S1', 'S2', 'S3', ..., 'S31'}; % All topics C = {'contrast1.img', 'contrast2.img', 'contrast3.img'}; For each = P = cell (31 * 3,1) for each topic, the required contrast images: Length = A for Jammu = 1: Length (C) P {j} = spm_select ('FPList', Fullfile (data_path, q {i}) sprint ('% s', cell2mat (c))); % Of each topic is to select three contrast images, it works in my script, it is probably not 100% correct, because I had to simplify this example. End of ...
Read more

python - Sequence Pattern recognition with Pybrain -

Image
I am constantly trying to use recurrent neural networks to classify a series of data. To be more specific, I have a sequence of sensor reading (which is continuous over time), I have to learn an algorithm which can detect the state related to this pattern, given the readings change. Example: Time step_1: 1.4 Time step_2: 1 Time step_3: 0.8 State = New Sequence: Time Phase 1: 0.4 Time Phase 2: 0.3 Time Phase 3: 0.1 State = Sitting I actually have 12 censors, I'm just showing a sequence of numbers for convenience. (Numbers are not real, I'm just trying to fulfill this idea)! I am trying to make my network with PyBrain RNN, however, I can not get a Data Set Container that can detect it, like information I tried to use sequential data But after some testing I came to know that this number is the next element in the sequence of numbers. Here's how I prepare my dataset: self.alldata = SequentialDataSet (ds.num_features, 1) # Now add samples to the dat...
Read more