Document production limits

[knz]

Fixes cockroachdb/cockroach#15771.

Fixes #1830.

@jseldess the way it's going to work is that I am going to iterate on this with Peter and Ben until we align on the technical content. Then I will hand this PR over to you so you can perform the necessary editorial fixes and cleanups.

[cockroach-teamcity]

This change is 

[cockroach-teamcity]

http://cockroach-docs-review.s3-website-us-east-1.amazonaws.com/a728ef9976a547f5b04f2786a4cfd64024e8a021/

[jseldess]

Sounds good to me, @knz. Thanks.

[bdarnell]

Review status: 0 of 1 files reviewed at latest revision, 18 unresolved discussions, all commit checks successful.

---

v1.1/limits.md, line 10 at r3 (raw file):

### Introduction

I think it would be good to provide a summary of the major limits before going into how they are derived.

---

v1.1/limits.md, line 29 at r3 (raw file):

a single node, CockroachDB needs:

- 64MiB on disk,

"Up to 64MiB on disk" (or maybe "Approximately"). It's rare for a range to be exactly full, and we store the data compressed (but split based on uncompressed size). On the other hand, rocksdb has some space amplification, so growing ranges will take up more space than their actual size until compaction occurs.

---

v1.1/limits.md, line 39 at r3 (raw file):

For example, in our test clusters with 8 cores per node, 16GB of RAM,
and unlimited disk space, we observe a practical limit of about 50.000

Write numbers in US style (commas not periods for thousands separators).

---

v1.1/limits.md, line 46 at r3 (raw file):

CockroachDB also supports changing the range size (e.g. to increase
it) but this is currently not fully tested. Different zones can use

If we're cpu-limited we should definitely test this. My estimate is that increasing range sizes will reduce CPU usage but increase memory.

---

v1.1/limits.md, line 76 at r3 (raw file):

of the tree is a single range called "meta1", containing descriptors
to "meta2" ranges, containing descriptors to the data ranges.  A range
descriptor is about 100-200 bytes, so theoretically with 64MB ranges

The size of a range descriptor is influenced by the application: indexing large string columns (for example) will result in larger range descriptors on average due to larger key sizes, and therefore lower overall capacity.

---

v1.1/limits.md, line 78 at r3 (raw file):

descriptor is about 100-200 bytes, so theoretically with 64MB ranges
meta1 can index ~550k meta2 ranges, and each meta2 range can index
~550k data ranges. This gives at the default range size a theoretical

Given the variability in range descriptor sizes, "theoretical" probably isn't the right word to use here.

---

v1.1/limits.md, line 79 at r3 (raw file):

meta1 can index ~550k meta2 ranges, and each meta2 range can index
~550k data ranges. This gives at the default range size a theoretical
maximum of 10PB capacity in a CockroachDB cluster.

The design doc does a similar computation with a result of 4 exabytes. We should compute this limit the same way in both places (updating the design doc if we now have more accurate information about range descriptor sizes).

---

v1.1/limits.md, line 84 at r3 (raw file):

limitation](https://github.com/cockroachdb/cockroach/issues/18032)
which prevents the creation of multiple meta2 ranges, so **until
CockroachDB v1.2 a cluster may become increasingly slower when using

Will it get slower? It should have similar consequences to increasing the max range size.

---

v1.1/limits.md, line 86 at r3 (raw file):

CockroachDB v1.2 a cluster may become increasingly slower when using
more than ~550k data ranges in total: clusters with a capacity of
about 1TB or beyond will not operate at max performance.** This is

550k * 64MiB is 33TiB (and because 550k is a limit on the number of ranges, not the number of replicas, we don't need to divide by 3). The 1.1 limit therefore is 33TiB of user data stored on 100TiB of raw disk, which must be spread across at least 30 nodes.

---

v1.1/limits.md, line 93 at r3 (raw file):

### Derived SQL schema limits

- Maximum number of databases = 2^63;

We can call 2^63 "unlimited".

However, IIRC databases also result in the creation of a range (which will always be empty), so they do run into the range limit. (Maybe we should stop auto-splitting ranges at schema id boundaries)

---

v1.1/limits.md, line 95 at r3 (raw file):

- Maximum number of databases = 2^63;
- Maximum number of tables = approx. maximum number of ranges or 2^63, whichever is smallest (but also see note below);
- Maximum number of indexes / FKs = approx. maximum number of ranges;

Indexes/FKs don't create their own range; the only limit for them is that they increase the size of their table descriptors.

---

v1.1/limits.md, line 97 at r3 (raw file):

- Maximum number of indexes / FKs = approx. maximum number of ranges;
- Maximum number of indexes / FKs per table = maximum number of indexes or 2^63, whichever is smallest;
- Maximum number of rows in a single table = approx. maximum number of ranges, times the number of rows per range;

This is only true if essentially all of your data is in one table. I think that phrasing it this way confuses more than it helps; we should just say that there is no limit on the size of a table except for the total limit on the data size of the cluster.

---

v1.1/limits.md, line 99 at r3 (raw file):

- Maximum number of rows in a single table = approx. maximum number of ranges, times the number of rows per range;
- Maximum number of rows per range = range size / minimum row size;
- Minimum number of rows per range = range size / maximum row size.

These last two are true, but are they useful?

---

v1.1/limits.md, line 108 at r3 (raw file):

  non-interleaved tables never share a range. This implies the maximum
  number of tables is limited by the maximum number of ranges.
- in CockroachDB v1.1, all the table and view descriptors (metadata)

Bring this comment up to the top of this section. It is the relevant limit in nearly all cases, and not the range count or 2^63.

---

v1.1/limits.md, line 112 at r3 (raw file):

  the cluster. Table/view descriptors can have a variable size,
  depending on the schema definition and the number of schema changes
  over time. Assuming a descriptor size of around 1K, the nominal

1KB seems high for a table descriptor, although some applications use very wide tables that will be even larger. Again, this is highly application-dependent.

---

v1.1/limits.md, line 117 at r3 (raw file):

  beyond the nominal maximum range size, albeit with a performance penalty.

### SQL Min/max row sizes

A row has two sizes: the size of the row itself (roughly equal to keyEncoding(pkColumns) * numColumnFamilies + valueEncoding(nonPKColumns)), which is the size that must be kept under the max range size, and the total amount of data that it contributes to the table's size (which includes the row itself plus secondary indexes: keyEncoding(idxColumns) + valueEncoding(pkColumns) + valueEncoding(storedColumns) for each index).

---

v1.1/limits.md, line 128 at r3 (raw file):

- the combined size of all primary key values in a single row should
  fit within a few kilobytes. Although CockroachDB might support keys

Kilobyte-sized primary keys would still be pretty bad. I'd recommend no more than 250 bytes for a primary key, with a strong preference for 8 or 16 bytes. (A few KB is a good guideline for fields used in secondary indexes)

---

v1.1/limits.md, line 133 at r3 (raw file):

  on each node, keys larger than a few kilobytes will cause a
  significant decrease in CockroachDB performance.
- the combined sized of a row must fit within a fraction of the range

We should also discuss the retention of MVCC versions, and that the combined size of a row includes all of its historical versions within the GC TTL (plus some factor to account for the fact that GC is not instantaneous when the TTL is reached).

---

Comments from Reviewable

[jseldess]

@knz, any plans to continue working on this? No rush, but I think it'd be great to have for 1.2. In conjunction, I'll be reworking our production deployment guidance.

[knz]

Yes I plan to work on this; however, the core team is currently making changes that heavily influence this document, and I'm waiting for the code to stabilize before I make the corresponding changes.

[jseldess]

@robert-s-lee, @vivekmenezes, @cuongdo, this is the fine-grained production limits document I referenced in our meeting earlier. Based on @knz's last comment, it seems like we need to wait from some core work before proceeding.

[jseldess]

@knz, would you mind adding some details about the core changes that will effect these limits? I discussed this briefly with @petermattis in the docs meeting today, and he wasn't sure what changes you might mean.

[jseldess]

@knz, @bdarnell, is this something we want to resurrect?

[knz]

This document is still important.

I was chatting with @nstewart last week and Nate highlighted the fact that giving an intuition about performance to users is really at the top of the list for sales (and I found this being stated ironic after one year of me harping on this subject, but hey, better late than never).

I am sorry I have not kept this PR up to date but there were a lot of changes in core and it's not directly my team, so it's hard for me to keep up. I think when we want to revive this I'd prefer to work hand-in-hand with someone with the best up-to-date knowledge.

[jseldess]

We have closed this issue because it is more than 3 years
old. If this issue is still relevant, please add a comment and reopen
the issue. Thank you for your contribution to CockroachDB docs!