Add operator.subscript as a convenience for creating slices

@gvanrossum

BPO	24379
Nosy	@gvanrossum, @rhettinger, @mdickinson, @vstinner, @taleinat, @ned-deily, @stevendaprano, @bitdancer, @vadmium, @serhiy-storchaka, @1st1, @MojoVampire, @llllllllll, @ilevkivskyi, @Vgr255, @shoyer
Files	slice.patch slicerepr.patch operator_subscript.patch operator_subscript_pyonly.patch operator_subscript_pyonly.patch operator_subscript_pyonly.patch operator_subscript_pyonly.patch operator_subscript_pyonly.patch operator_subscript_pyonly.patch operator_subscript_norefleak.patch operator_subscript_norefleak_v2.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2018-06-16.02:02:00.835>
created_at = <Date 2015-06-04.06:07:04.227>
labels = ['extension-modules', 'type-feature', '3.7']
title = 'Add operator.subscript as a convenience for creating slices'
updated_at = <Date 2018-07-20.00:08:42.343>
user = 'https://github.com/llllllllll'

bugs.python.org fields:

activity = <Date 2018-07-20.00:08:42.343>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2018-06-16.02:02:00.835>
closer = 'rhettinger'
components = ['Extension Modules']
creation = <Date 2015-06-04.06:07:04.227>
creator = 'llllllllll'
dependencies = []
files = ['39614', '39615', '39775', '39792', '39902', '39904', '39906', '39907', '39913', '45471', '45473']
hgrepos = []
issue_num = 24379
keywords = ['patch']
message_count = 60.0
messages = ['244801', '244802', '244804', '244805', '244811', '244819', '244820', '244821', '245070', '245072', '245082', '245131', '245133', '245134', '245657', '245665', '245666', '245702', '246618', '246624', '246625', '246631', '246634', '246635', '246637', '246638', '246752', '247286', '247528', '247530', '248703', '248705', '248706', '248708', '248710', '253291', '253894', '253899', '253902', '254031', '256895', '272773', '272774', '272863', '280456', '280639', '280648', '280661', '280692', '280699', '280700', '280703', '280706', '280707', '280716', '280721', '319507', '319695', '321966', '321972']
nosy_count = 18.0
nosy_names = ['gvanrossum', 'rhettinger', 'mark.dickinson', 'vstinner', 'taleinat', 'ned.deily', 'Arfrever', 'steven.daprano', 'r.david.murray', 'python-dev', 'martin.panter', 'serhiy.storchaka', 'yselivanov', 'josh.r', 'llllllllll', 'levkivskyi', 'abarry', 'Stephan Hoyer']
pr_nums = []
priority = 'normal'
resolution = 'rejected'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue24379'
versions = ['Python 3.7']

I often find that when working with pandas and numpy I want to store slice objects in variables to pass around and re-use; however, the syntax for constructing a slice literal outside of an indexer is very different from the syntax used inside of a subscript. This patch proposes the following change:

slice.literal

This would be a singleton instance of a class that looks like:

class sliceliteral(object):
    def __getitem__(self, key):
        return key

The basic idea is to provide an alternative constructor to 'slice' that uses the subscript syntax. This allows people to write more understandable code.

Consider the following examples:

reverse = slice(None, None, -1)
reverse = slice.literal[::-1]

all_rows_first_col = slice(None), slice(0)
all_rows_first_col = slice.literal[:, 0]

first_row_all_cols_but_last = slice(0), slice(None, -1)
first_row_all_cols_but_last = slice.literal[0, :-1]

Again, this is not intended to make the code shorter, instead, it is designed to make it more clear what the slice object your are constructing looks like.

Another feature of the new literal object is that it is not limited to just the creation of slice instances; instead, it is designed to mix slices and other types together. For example:

>>> slice.literal[0]
0
>>> slice.literal[0, 1]
(0, 1)
>>> slice.literal[0, 1:]
(0, slice(1, None, None)
>>> slice.literal[:, ..., ::-1]
(slice(None, None, None), Ellipsis, slice(None, None, -1)

These examples show that sometimes the subscript notation is much more clear that the non-subscript notation.
I believe that while this is trivial, it is very convinient to have on the slice type itself so that it is quickly available. This also prevents everyone from rolling their own version that is accesible in different ways (think Py_RETURN_NONE).
Another reason that chose this aproach is that it requires no change to the syntax to support.

There is a second change proposed here and that is to 'slice.__repr__'. This change makes the repr of a slice object match the new literal syntax to make it easier to read.

>>> slice.literal[:]
slice.literal[:]
>>> slice.literal[1:]
slice.literal[1:]
>>> slice.literal[1:-1]
slice.literal[1:-1]
>>> slice.literal[:-1]
slice.literal[:-1]
>>> slice.literal[::-1]
slice.literal[::-1]

This change actually affects old behaviour so I am going to upload it as a seperate patch. I understand that the change to repr much be less desirable than the addition of 'slice.literal'

Here is the patch that includes the updates to 'slice.__repr__'

FWIW, I like this idea.

Why not index the slice type itself? slice[1:2]

Another feature of the new literal object is that it is not limited to just the creation of slice instances; instead, it is designed to mix slices and other types together.

This looks as disadvantage.

I'm with Serhiy, I don't think we need a "literal", just make slice itself indexable:

reverse = slice(None, None, -1)
reverse = slice[::-1]

The only question in my mind is what slice should do when given just a single index:

slice[0]

I suppose that should be a ValueError?

For prior art, it's worth taking a look at NumPy, and in particular its s_ and index_exp functions:

>>> import numpy as np
>>> np.s_[1:2]
slice(1, 2, None)
>>> np.s_[0]
0
>>> np.s_[1:2, 3]
(slice(1, 2, None), 3)

(Correction: they're not functions, or even callables.)

Why not index the slice type itself? slice[1:2]

I originally considered this and I personally really like this syntax, but I was concerned with ambiguity with the typing module

The only question in my mind is what slice should do when given just a single index

I think some of the power of this concept comes from the fact that I can express a complicated indexer without worrying about how it desugars. I would personally prefer being able to have this return tuples and scalars so that the syntax is easier to explain.

(This should probably be discussed on the Python Ideas mailing list...)

I definitely like the idea of being able to construct slices etc. using "[]" syntax. I think this should be considered even if it is decided not to change the repr() of slices.

An important note here is that this is about more than slices:

$ python3
Python 3.4.2 (default, Feb 23 2015, 21:16:28)
[GCC 4.2.1 Compatible Apple LLVM 5.1 (clang-503.0.40)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> class A:
...     def __getitem__(self, *args):
...         print(repr(args))
...
>>> a = A()
>>> a[0]
(0,)
>>> a[0:1]
(slice(0, 1, None),)
>>> a[0:1, ..., 1:2]
((slice(0, 1, None), Ellipsis, slice(1, 2, None)),)
>>> a[0:1, 2]
((slice(0, 1, None), 2),)

Indeed, Joe's suggested slice.literal supports all of this, but we can't just modify slice to handle all of these cases.

What I'm missing is a way to use such an object to actually index/slice something. The only way I can currently think of is using a.__getitem__(), but that's quite ugly IMO.

What I'm missing is a way to use such an object to actually index/slice something

Sorry, I am not sure I understand what you mean by this? You can pass a slice object, or a tuple of slices in subscript notation.

>>> [1, 2, 3, 4][slice(2)]
[1, 2]

Also, I am not currently subscribed to python-ideas; what is the common practice for posting new threads?

(see https://mail.python.org/mailman/listinfo/python-ideas)

But for x = [1,2,3,4], how will x[y] work for all of the following values of y?

y = slice.literal[0]
y = slice.literal[1:2]
y = slice.literal[0:1, ..., 3]

NumPy's s_ "magic object" is a factory, returning objects of different types depending on the given input. If you want an actual slice.literal class, then you'll have to supply a way to convert it into an object usable for indexing/slicing, e.g.:

y = slice.literal[0:1]
[1,2,3,4][y.as_indexer]

>>> slice.literal[0]
0
>>> y = slice.literal[1:2]
slice(1, 2, None)
>>> slice.literal[0:1, ..., 3]
(slice(0, 1, None), Ellipsis, 3)

The way this object works right now does not create instances of some inner class of slice, instead, indexing it returns the key without modification.

So, is this in any ways different than NumPy's s_?

Serhiy, thank you for review! Here is a corrected patch.

operator_subscript_norefleak_v2.patch LGTM. Waiting for Ned's decision about 3.6.

I don't think this is appropriate for 3.6 now. We're a little more than 4 weeks from the final release - way past the feature code cut-off - and, from the comments here, there is no longer a total consensus that this feature should go back in at all after being pulled and largely forgotten about for a year.

Maybe then it makes sense to apply this to 3.7? It looks like most people are in favour of this (also on python-ideas), only Raymond is not sure/mildly against.

Actually I'm with Raymond -- I'm also not sure or mildly against (-0).
Given how easy it is to do without and how cumbersome using the operator
module is I don't see a great benefit. (IMO the operator module should be
the measure of *last* resort -- it is not to become part of anybody's
toolkit for common operators. But I seem to be a minority opinion here.)

So 3.8 then, or should this be closed?

FWIW I'm -1. IMO this should be a code recipe somewhere, no need for it to be in the stdlib.

Let's close it. Just because someone spent a lot of effort on a patch we don't have to accept it.

Raymond, Tal and Guido -- do any of you work routinely with multi-dimensional arrays?

In my experience as someone who uses Python everyday for numerical computing (and has been doing so for many years), the need for an operator like this comes up with some regularity. With multi-dimensional indexing, this allows lets you cut down quite a bit on boilerplate, e.g., compare

subscript[:, 0, ::-1] vs (slice(None), 0, slice(None, None, -1))

It's absolutely true that subscript is an easy four line recipe, and indeed a form of this recipe is already included in both NumPy and pandas. But I think this is a case where the lack of a common idiom is actively harmful. I do multi-dimensional indexing in using at least four different libraries (pandas, xarray, numpy and tensorflow), and it feels wrong to use a utility from numpy/pandas for other projects. I could write my own version of operator.subscript in each project (and yes, I've done so before), but for any individual use case it's less hassle to write things out the long way.

In practice, the preferred way to write such long expressions seems to be to redundantly repeat indexing operations, e.g.,

x[:, 0, ::-1]
y[:, 0, ::-1]

rather than

  index = subscript[:, 0, ::-1]
  x[index]
  y[index]

This is definitely non-ideal from a readability perspective. It's no longer immediately clear that these arrays are being indexed in the same way, and any changes would need to be applied twice.

Raymond, Tal and Guido -- do any of you work routinely with multi-dimensional arrays?

Hi Stephan Hoyer,

IMHO a *closed* issue is not the most appropriate place to request Guido to change his mind. You may get more traction on python-ideas where you may find more supporters who need the operator.

The question is not only if the operator makes sense (obviously, it does, for you), but if it "belongs" to the standard library. Is it popular enough? Is it consistent with other functions of the operator module? etc.

Navigation Menu

Repository navigation

Add operator.subscript as a convenience for creating slices #68567

47 remaining items

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

📋 Server Request & Proxy Logs

⚙️ Pengaturan Proxy Target

Add operator.subscript as a convenience for creating slices #68567

Description

Activity

llllllllll commented on Jun 4, 2015

llllllllll commented on Jun 4, 2015

rhettinger commented on Jun 4, 2015

serhiy-storchaka commented on Jun 4, 2015

stevendaprano commented on Jun 4, 2015

mdickinson commented on Jun 4, 2015

mdickinson commented on Jun 4, 2015

llllllllll commented on Jun 4, 2015

taleinat commented on Jun 9, 2015

llllllllll commented on Jun 9, 2015

taleinat commented on Jun 9, 2015

llllllllll commented on Jun 10, 2015

taleinat commented on Jun 10, 2015

47 remaining items

ilevkivskyi commented on Nov 13, 2016

serhiy-storchaka commented on Nov 13, 2016

ned-deily commented on Nov 13, 2016

ilevkivskyi commented on Nov 13, 2016

gvanrossum commented on Nov 13, 2016

taleinat commented on Jun 14, 2018

gvanrossum commented on Jun 16, 2018

shoyer commented on Jul 19, 2018

vstinner commented on Jul 20, 2018

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions